DOCUMENT RESUME 



ED 464 3.85 



EA 031 589 



AUTHOR 

TITLE 

PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



Kraft, Nancy P. 

Critical Characteristics of Successful After-School 
Programs: An Evaluation of the 21st Century Initiative. 
2001-04-00 

3 6 p . ; Paper presented at the Annual Meeting of the American 
Educational Research Association (Seattle, WA, April 10-14, 
2001 ) . 

Reports - Research (143) -- Speeches/Meeting Papers (150) 

MF01/PC02 Plus Postage. 

* Academic Achievement; * After School Programs; Behavioral 
Objectives; Educational Change; ^Educational Improvement; 
Elementary Education; Parent School Relationship; Public 
Schools; School Effectiveness; Social Values 



ABSTRACT 



With the recent U.S. Department of Education's 21st Century 
Initiative there has been a proliferation of after- school programs emerging 
in rural, suburban, and urban school communities across the country. Although 
these programs are financially supported through the federal government, the 
grants are only funded for a 3 -year period. Given the short duration of the 
federal support for these programs and the fact that education innovations 
generally take time to become established and well -running programs, this 
research sheds light on what variables are needed for successful after-school 
programs. The document reports the results of an evaluation that was 
conducted on five after- school programs in northeastern Kansas. Its purpose 
was to present a context and a composite picture of the after- school program, 
to establish baseline data against which the programs can be evaluated over 
the course of the 3 -year project, to identify best practices and 
characteristics of quality after-school programs in the region, to facilitate 
a continuous monitoring of the program and suggest areas where improvements 
might be made to enhance program quality, and* to determine the impact the 
program had on participants and to what degree program objectives had been 
realized. (Contains 16 references and 22 endnotes.) (DFR) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 






Critical Characteristics of Successful After-School Programs: 
An Evaluation of the 21 st Century Initiative 



10 

00 

m 



VO 



Q 



m 



Nancy P. Kraft 
Assistant Professor 
University of Kansas 
nkraft@ukans.edu 



BEST COPY AVAILABLE 



American Educational Research Association Annual Meeting, Seattle, WA, April 2001 



<T- 

0o 

U) 



C*0 

O 




U S DEPARTMENT OF EDUCATION 
Oftice of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
/ CENTER (ERIC) 

m This document has been reproduced as 
received from the person or organization 
originating it. 



□ Minor changes have been made to 
improve reproduction quality. 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 

N. P. Krgft 



° Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



2 



Critical Characteristics off Successful After-School Programs: 
An Evaluation of the 21 st Century Initiative 



Introduction to the Study 

With the recent U.S. Department of Education’s 21 st Century Initiative there has been a proliferation of 
after-school programs emerging in rural, suburban, and urban school communities across the country. 
While these programs are financially supported through the federal government the current duration of the 
grant is for a three-year period of time. Given the short duration of the federal support for these programs 
and the fact that starting any new educational innovation generally takes time to become an established and 
well-running program, this research sheds light on what variables are needed for successful after-school 
programs. It also raises a number of questions concerning the dilemmas encountered in trying to implement 
quality programs that have a short funding cycle - putting energy into ensuring excellence or addressing 
sustainability, and which focus should takes precedence. While the government’s absolute priority goal for 
programs funded during the 1999/00 school year was to expand learning opportunities for children and 
youth in the community, this research reveals contradictions between stakeholder groups in what this 
means. This research also makes a contribution to the “after school field” in addressing one of the biggest 
challenges facing after-school programs today - identifying the key ingredients of a quality program. 

This study reports the results of an evaluation that was conducted on five after-school programs in North- 
east Kansas. The purpose of the evaluation was to 1) develop a context and composite picture of the after- 
school programs; 2) establish baseline data against which the programs can be evaluated over the course of 
the three year project; 3) identify best practices and characteristics of quality after-school programs in the 
region; 4) facilitate a continuous monitoring process for the program and suggest areas where improve- 
ments might be made to enhance program quality; and 5) determine the impact the program had on partici- 
pants and to what degree program objectives had been realized. 

Theoretical Framework Grounding the Study 

For all 21 st Century Community Learning Centers, the federal government requires that a “strong evalua- 
tion plan that will shape the development of the project from the beginning of the grant period,” is included 
in the proposal to secure funds for these programs. 1 The REP is very specific in detailing what a compre- 
hensive evaluation should include, i.e., the types of data that will be collected, when data will be collected, 
the design, methods, and instrumentation used to collect data, how the data will be analyzed, and when and 
how reports of results and outcomes will become available and used. Ironically, the formal evaluation 
procedures requested by the federal government for reporting and accountability purposes of the 21 st Cen- 
tury Community Learning Centers initiatives are grounded in a very technical measurement approach 
primarily based dh gathering test score and performance data. This “performance report” primarily consists 
of reporting outcomes data such as student achievement test scores, teacher grades in the core content areas 
(i.e., math and reading), which for the most part are arbitrary, and attendance and behavior data (which 
again basically consist of arbitrary teacher data). Instead what is needed, as identified by Seligson, founder 
of the National Institute on Out-of-School Time, are 

studies that.. .look at the whole ecology of the program - the quality, the activities, etc.- 
not just at educational outcomes. The definition of desired outcomes should be broad so 
that one does not look only at test scores, but also the social and emotional development of 
the child, which is tantamount to doing well in school and in life. 2 
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Thus, the theoretical framework guiding this study (and the evaluation plan written into this successfully 
funded program) was the CIPP Model which is based on the view that the most important purpose of an 
evaluation process is not to prove but to improve. 3 Unlike first generation models of evaluation that focus 
on measurement issues or second generation models which describe patterns of strengths and weaknesses in 
respect to specific objectives, the C1PP model, one of several third generation evaluation models was used. 4 
While associated with making judgments concerning the program under evaluation, the CIPP framework 
focuses on decision-making processes that need to occur at all levels of program implementation and in a 
formative sense, facilitates program improvement, which is one of the major goals of evaluation of after- 
school programs. 5 The CIPP evaluation model elements are context, input, process, and product. Given the 
short three-year funding cycle of this program and the fact that local sustainability efforts would be based 
on the presumed effectiveness of the program, it was important to use a comprehensive evaluation process 
that would provide insight into all aspects of the program - from the conception of the after-school pro- 
gram model to the actual implementation at each of the sites. 

In the CIPP model, context is the institutional context or setting within which decisions are made. In 
studying context one also examines “whether existing goals and priorities are attuned to the needs of who- 
ever is being served.” 6 Especially crucial, as in the case of the after-school programs focusing this research, 
was to study the context and the unique set of variables in each school/community that either served to 
support or create a barrier to successful implementation. Input in the CIPP model identifies and assesses 
the system capabilities, strategies, and procedural designs for implementing the program. Thus it was 
important to look at the program as a whole and to also look at the unique variables at each of the five 
sites, human and non human variables, that either facilitated or inhibited implementation of the program 
model. 

Addressing the process component allows for decisions concerning how well the implementation of the plan 
is actually occurring, or the fidelity of the model to the actual program that is being implemented. It was 
critical to determine to what degree the various program components were actually being implemented at 
each individual site and then to understand “why” if the program was not being implemented as defined. A 
determination of the degree of program fidelity facilitated decisions for professional development and 
program improvement during the second year of operation. Finally, product evaluation measures, inter- 
prets, and judges pro- gram outcomes and determines to what extent the program has met the needs of the 
clients it was intended to serve. 

Whereas many evaluation schemata look at individual aspects of the program and often use measurement 
models of evaluation, the CIPP model enabled a system or “holistic” view of the whole ecology of the 
program. Using a holistic approach, such as the CIPP model, facilitated an indepth examination of these 
after-school programs that went far beyond looking only at outcomes. This study provided valuable data, in 
a formative sense, that was subsequently used during the second year of the program’s operation as a 
means to improve the existing program and address the issues raised during the initial evalu- ation process. 
Unlike traditional third generation models of evaluation that posit the evaluator in a judgmental role 
regarding the programs merit and worth, the process of inquiry used in this study was also grounded in 
principles of responsive evaluation in which the issues, concerns, and views of multiple stake- holders were 
taken into consideration. 7 Stakeholders were given opportunities to define the issues and questions they felt 
critical to address in the context of their own site. The evaluation process, itself, involved multiple stake- 
holders (i.e., parents, students, teachers, principals, site coordinators and staff, the program coordinator, 
community volunteers, and members of the advisory committee) in co-constructing a picture of what these 
Community Learning Centers looked like and defining to what degree the varying needs were addressed. 
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Methods of Inquiry 

Methodology guiding this research was grounded in qualitative methods and based on naturalistic inquiry 
principles . 8 Among these principles is how one views the nature of reality. Contrary to positivism that 
asserts that reality is single, tangible, and fragmentable, this researcher embraced the notion that realities 
are multiple, constructed, and holistic. When a program involves and impacts multiple stakeholders, each 
brings to bear on the evaluation process their perceptions and interpretations of reality. The role of the 
evaluator becomes one of sifting and sorting through these multiple versions of reality to make sense or 
apply meaning to the phenomenon under study. Another principle addresses the relationship between the 
knower and the known. Choices and decisions are made throughout the entire evaluation process - which 
stakeholder groups to include, which to survey, what to observe (and what gets observed and what does 
not), which stakeholder groups should be interviewed, what questions to ask, what kinds of follow-up 
questions are required, when to probe for further information, etc. Given this fact, to assert there is inde- 
pendence between the knower and known (as is the case with positivism) seems absurd in that the knower 
and known are interactive and inseparable. While there are ways to ensure checks and balances during the 
evaluation process, to ensure trustworthiness of the data, the fact remains that objectivity in its pure form is 
an unattainable state. And finally the role of values. While positivist believe that evaluation can and should 
be value free, this evaluator believes that evaluation is value-bounded. When one evaluates programs 
against standards of quality - one must realize that values determine and define what is quality. When an 
evaluator is judging program effectiveness one looks for characteristics that exemplify effectiveness; again 
those characteristics of effectiveness are often the result of what one values to be effective. Rather than 
dismissing values, it is more important to acknowledge the influence that values have. Lincoln and Guba 
contend, “Such a course is infinitely to be preferred to continuing in the self-delusion that methodology can 
and does protect one from their unwelcome incursions !” 9 

Multiple methods were used because according to Denzin and Lincoln, “the use of multiple methods, or 
triangulation, reflects an attempt to secure in-depth understanding of the phenomenon in question. Objective 
reality can never be captured .” 10 While data collection methods were primarily qualitative in nature, 
quantitative data were also collected. These included: 

• Standardized state assessment instruments, district assessment/evaluation tools, and pre and post 
curricuiuui-based measurements. These data were used primarily to track academic growth, 
discipline referrals>attendance, and numbers of activities in which children participated. 

• Surveys of teachers, parents, community volunteers, student assistants, and after-school staff 
ascertaining their perceived value of the after-school program and their perceptions of the impact 
the program had on children and the communities in which they were located. In addition to Likert 
scale and rank ordering items, surveys also asked open-ended questions. Parents and teachers were 
surveyed at mid-year to ascertain their beliefs about, and satisfaction with the program and to offer 
suggestions for program improvement. They were surveyed again at the end of the year. 

• Surveys of site coordinators examining to what degree their programs aligned with the designated 
program model and determining what value coordinators placed on each of the program compo- 
nents. They were also asked their opinions about the coordination and administrative functions of 
the program director and were asked to critically self-reflect on their own roles as site coordinators. 

• At midyear an attempt was made to collect data that would give insight into students’ level of 
resiliency, but because of privacy concerns voiced by several of the site coordinators, the survey 
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wasn’t completed by all sites. 



Qualitative methods included: 

e Focus group interviews with advisory committee members, program staff, and select parents to 
examine the value and the degree of success the program had during its first year of operation. A 
pre- and post- focus group was conducted with teachers at one of the program sites - prior to the 
beginning of the program to learn their issues and concerns and to get input into program design 
and again at the end of the first year to determine if their issues and concerns had been addressed 
and to hear their reactions and responses to the program. A focus group with site coordinators was 
also conducted mid-year to determine the progress being made with the programs and to learn 
about needed changes. 

• Individual interviews were conducted with students, parents, principals and the program director. 
Students were asked what they liked and did not like, about the program, why they attended the 
program, and what benefits they felt the program had on them, as students. In addition to examin- 
ing the value and benefits of the program, principals were also asked to define what they believed 
principal support looked like. The program director went through an extensive interview that 
addressed all aspects of the program - from planning to implementation, and her perceptions of the 
value, benefits, and concerns for succeeding years. 

Both the focus groups and individual interviews were transcribed. A content analysis was con- 
ducted to examine patterns and themes that emerged from the data in the interview transcripts. 

• Non-participant observation consisted of fifteen 3-hour observations that included three separate 
visits per site with the first visit occurring in September-October 1999, the second visit during 
January-February 2000, and the third visit during May, 2000. The purpose of the site observations 
was to develop a composite picture of what each CLC program looked like, to observe how 
individual sites were operationalizing the program model, and to talk informally to staff, parents, 
and children attending the program. All observations used a standard observation form that seg- 
mented the observation into blocks of time and recorded activities teachers and children were 
engaged in. Anecdotal data supplemented the actual observation. 

An analysis of the observations was facilitated by the National School-Age Care Alliance Stan- 
dards that list quality standards for after-school programs addressing human relationships; indoor 
and outdoor environment; safety, health, and nutrition; and administration program indicators. 11 

Research Questions Guiding the Evaluation 

Table I presents the original framework that guided the evaluation listing the evaluation questions, the 

indicators, and data sources. Minor changes were made throughout the year. 
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Table 1 - Evaluation Framework 



Evaluation Questions 
(what we want to know) 


Key Indicators 
(what we are measuring) 


Data Sources/Methods 
(how we are measuring it) 


1. What educational benefits 
have occurred as a result of the 
CLCs? 


Performance on the Kansas state 
assessments 

Indicators of reading achievement 
Indicators of math achievement 


State assessment data 
Standardized test scores 
Pre-post and continuous curriculum 
based assessment in reading and math 


2. What social benefits have 
occurred as a result of the CLCs? 


Indicators of increased resiliency, e.g., in- 
sight, independence, relationships, initia- 
tive, creativity and humor, morality (the 
difference between right and wrong), and 
general resiliency 


Student interviews 

Parent interviews and focus group 

Teacher surveys 

Site coordinators surveys 


3. What behavioral changes are 
occurring in children as a result 
of the program? 


Indicators of decreased incidences of vio- 
lence and discipline referrals 
Indicators of improved student attitudes to- 
ward school 


Teacher survey 
Student interviews 
Parent survey 
Site coordinator survey 


4. To what degree do the CLCs 
provide youth/community mem- 
bers enriched learning opportuni- 
ties in a safe, healthy, and drug- 
free environment? 


Reports of activities 

Parent, community, student perceptions 


Student interviews 
Community volunteer surveys 
Parent volunteers 


5. What types of educational, 
developmental, and recreational 
services do the CLCs offer to 
enhance student resiliency? 


Indicators of activities that are planned and 
implemented 


Student activity sign-in sheets 
Matrix of activities assessing relevance 
to resiliency framework 


6. To what extent are the stu- 
dents most at-risk of academic 
failure being served? 


The number of at-risk student attendees 
Indicators of increased attendance patterns 
of children scoring below the 50 th percentile 
and rated as most in need of participation 


Teacher referral lists 
Student test score data 
Student activity sign-in sheets 


7. What is the fidelity of local 
CLCs to the established program 
design? Is the model being im- 
plemented? 


Indicators of how closely the model is fol- 
lowed - the discrepancy between the “ideal” 
and the “real” 


Characteristics of implementation check- 
list 

Site Coordinator survey 
Observations of program implementation 


8. How does the local program 
model compare to criteria for 
quality after-school programs? 


Quality indicators of after-school programs 
Federal guidance for after-school programs 


After-School Program staff surveys 
and/or checklists from the Nat’l School- 
Age Care Alliance 

21 st Century after-school programs sur- 
veys 


9. How do the CLCs support 
schools to become centers of the 
community (opening the schools 
during out-of-school hours to 
children, parents and the com- 
munity)? 


Indicators of operation of the after-school 
program, e.g., hours, days, months 
Indicators of participation of children, par- 
ents, and community members 


Checklist of operation procedures 
Focus group with community 
Interviews with parents 
Activity sign-in sheets 
Rosters of parents and community mem- 
bers 



0 
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Evaluation Questions 
(what we want to know) 


Key Indicators 
(what we are measuring) 


Data Sources/Methods 
(how we are measuring it) 


10. How do the CLCs support 
collaboration with school staff 
and community organizations to 
provide and sustain services to 
the community. 


Indicators of quality interactions between 
CLC staff and school staff, community 
members, and parents 

The number of parent/community volunteers 
involved with the CLCs 


CLC staff, teacher, parents, community 
members focus groups 
Staff documentation of collaboration 
Survey of staffteacher collaboration 
Survey of community members and par- 
ent collaboration; observations 


11. What is the degree of satisfaction with the CLCs... 


a) from parents? 

b) from students? 

c) from classroom teachers? 

d) from CLC staff? 


Indicators of social validity 
Perceptions of staff in the program 


Parent surveys and interviews 
Student interviews 
Teacher surveys and focus group 
CLC staff surveys and focus group 



In a participatory evaluation mode, each program site was also given an opportunity to generate question(s) 
that they wanted answered at the end of the first year. Questions emerged from local Advisory Council 
meetings. Three of the five sites generated additional questions as part of the overall evaluation process. 
These questions are beyond the purview of this paper, which is an evaluation study conducted on the 
general program. 

Data Sources 

Data sources included all children (n=l 10) and parents (n= 100/response rate 40%) of children who were 
enrolled in the after-school programs. Also included were elementary teachers from each of the participa- 
ting schools (n=40/response rate 63%) and elementary principals (n=6/response rate five of six) from each 
school district. Program staff at each site ranged from three to five employees (dependent on the number of 
children served). These included one program coordinator per site, paid staff assistants, paid student 
helpers, and local community volunteers. The project director also provided data about the program. 

Context of the After-School Program 

In the spring of 1999 the Northeast Kansas Education Service Center (NEKESC) was successful in 
receiving funding through the U.S. Department of Education’s 21 st Century Initiative to establish Commu- 
nity Learning Centers (CLS’s) or after-school programs in elementary schools in five school districts: 
Effingham, Jefferson County North, McLouth, Perry-Lecompton, and Valley Falls. 

The model for the Northeast Kansas Community Learning Center (NEKCLC) was based on resiliency 
research and social development theory and was designed around three major components that included: 

1) a focus on strengthening academic skills; 

2) planned group time focused on building healthy relationships (the NEKCLC was a pilot 
site for a project sponsored through the Kansas Health Foundation involving the receipt of 
curriculum and technical assistance from Foundations, Inc., of Philadelphia); and 

3) opportunities for widening students’ horizons. 




- 6 - 



A fourth component that focused on establishing Parent Resource Centers was also built into the plan. 

Each school also differed a little from each other concerning the ages of children attending the program, the 
physical site that housed the program, and the kinds of activities in which children participated. 

While schools had individual autonomy in determining their hours of operation. Centers, for the most part 
were open between 3:00 and 6:00 p.m. Monday through Friday, everyday that school was in session. Built 
into the model was centralized coordination, administration, and supervision carried out by a program 
director with local control of specific optional activities at the individual school CLC level under the 
direction of site coordinators. 

Table III provides contextual information about each program’s attendance rate, hours of operation, when 
the programs began operation, and transportation information. 



Table HI - Site Information 


School 


Total Num- 
ber Served 


Avg Daily 
Attendance 


Hrs of Opera- 
tion 


Starting 

Date 


Transportation 


Effingham 


15 


11 


3:00-5:30 


9/20/99 


None 


JCN 


24 


16 


3:40-5:30/6:00 


9/13/99 


Bus return to 
Winchester 


McLouth 


26 


20 


3:15-5:30/6:00 


9/27/99 


Provided 


Perry- 

Lecompton 


31 


25 


3:20-5:30/6:00 


10/25/99 


Bus to and from 
Lecompton 


Valley Falls 


48 


41 


3:20-5:45/6:00 


9/27/99 


Provided 



Evaluation Findings 

The following summaries and discussions of the data are organized by the key questions focusing the 
evaluation. Based on the discussions, recommendations for program improvement are inserted throughout 
the evaluation report. 

I. NEKCLC Program Benefits 

Benefits of the program will be examined from the perspective of educational and social/behavioral changes 
observed in children. 

A. Academic Benefits 

In addition to collecting data from all stakeholders in the program, student grades were also used as an 
indicator in determining academic benefits. 

(1) Teachers’ perceptions. Teachers responded to two separate surveys over the course of the year - 
one in December or mid-year and one in May at the close of the first year. Of the 37 respondents in the 
mid-year evaluation, 62% or nearly 2/3 of the teachers reported the program as being academically 
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beneficial with responses varying from benefiting slightly to very effective. Teachers noted an increase in 
spelling skills, tests scores, and math. They indicated that it was obvious that the students were practicing 
phonics, spelling, and math facts. Typical comments included: test scores are improving, children are 
gaining more confidence, coming better prepared to class, completing assignments, and are more sure of 
themselves. Five noted that the after-school program did not seem to be making any difference academically 
with one commenting that the program seemed to be “more social than academic.” 

At the end of the year when teachers were asked if the after-school program was a worthwhile activity, 
27/30 said yes with three indicating no. Concerning the academic impact the program had on students, the 
vast majority indicated that it had a positive academic impact. Typical comments included: both students 
mastered fluency for 2 nd grade level - passed ORL’s and most outcomes, I feel both students benefited 
with extra practice in reading, homework was done and brought back, it enhanced reading comprehension, 
and those students who attended were able to complete what they did in CLC with high quality or positive 
results. There were some, however, who said that it did not have much of an impact, or gave statements 
such as, “while my students improved it is hard to tell if it was due to CLC.” 

Teachers often tied academic impact to the quality of the homework and the rate of homework completion. 
In the mid-year evaluation 1 8/37 (just less than 50%) responded positively about homework completion. 
Comments ranged from the quality is generally good and accurate to great quality and excellent. Ten 
commented neither positively nor negatively, stating such things as students don’t have homework to 
complete, more one-on-one is needed and should be checked more closely. Five had negative comments 
about homework, stating that it continues to be rushed, the quality is poor, and not a great change. 

Responses were rather similar in the final evaluation. Positive comments included such things as the 
homework was done neatly and in a timely manner, excellent, above average, or it increased the return of 
quality homework. But a large number of teachers, who seemed to think that the primary purpose of the 
CLC was to provide assistance with homework, shared their frustrations in comments such as that of one 
teacher who said, “For our students the academic instruction time to complete homework was at the very 
end of the CLC time. Many of our students were picked up early or were not very efficient at completing 
assignments.” Another teacher who believed that the CLCs should focus more on homework stated, “Older 
students could have benefitted from more time for work completion. Too much enrichment at times.” 

(2) Principals’ perceptions. All principals interviewed thought that their after-school program had 
significant academic benefits as well. One principal commented 

there is academic support particularly for kids who were at-risk that there wouldn’t be any 
support at home. When we give the work here we know it gets done, even if it’s just 
another 20 minutes of reading practice, you know it got done. 

While one indicated that she had not “looked specifically at after-school students to see if they have 
increased their grades or their test scores on anything,” she also said that none of the teachers had com- 
plained about some of the students attending the program, academically. As a way to evaluate the effective- 
ness of the academic benefits on the program, this principal said that she would like to see students’ 
curriculum-based measurement (CBM) scores next year to see the academic progress that students were 
making. Another commented that “there is no way to measure it exactly,” but then added that he felt what 
was being done in the CLC program was reinforcing state required outcomes. One principal believed that 
the direct instruction component should continue to be emphasized because any opportunity to help children 
become better readers was time well spent. 



O 

ERIC 



- 8 - 



In general, however, comments that focused on the academics also led to discussions about homework with 
one indicating that some parents and teachers wanted more time spent on homework. Another regarded the 
academic portion of the program as providing those students who were in the 3 rd , 4 th , or 5 th grades with 
homework assistance. This same principal voiced a concern about increasing student enrollment and how 
she believed that might be detrimental to the academic value of the program. In querying what the purpose 
of the after-school program was, one principal commented, 

I think some teachers viewed it as the homework warehouse, where they send their home- 
work and it needs to be completed there and if not, then the program’s not doing what it’s 
supposed to do. And from what I’ve heard and what I perceive, that’s not necessarily what 
the program was intended to do, it’s not a place where homework is sent and necessarily is 
to be completed. I mean there are other things that are the focus of the program, such as 
exposure to the arts and crafts, and the curriculum that they were using. I mean there is 
more to it then just going over to a study hall, and 1 think some teachers view it that way. 

(3) Program/site coordinators’ perceptions. When asked their perceptions of the educational benefits 
of the program, four of the five site coordinators’ responses were connected to homework activities, men- 
tioning specifically that it provided the necessary time to give their children assistance in homework com- 
tion. Other educational benefits included extra time for reading, direct instruction, or constant testing of 
basic math facts. One person shared that “the collaboration between staff and classroom teachers bridges 
the gap that may happen from school to home.” One other site coordinator added an additional education 
benefit of exposing the children to good literature. 

In reflecting on the educational benefits and the perceived relationship between academic achievement and 
the preponderance of homework, the program coordinator believed, for the most part, that the CLCs should 
include a strong homework component. She commented, 

I’m glad that since the kids have, in some places, an inordinate amount of homework that 
we are able to give it. I think it has been valued by the parents, it has been valued by the 
teachers, it has been - however, we spend so much time on it, for us. ...I don’t like seeing a 
first grader with 30 minutes of homework and we have that in some schools. 

She surmised, 

I wonder if they think they (the children) go to the after school program - they can get it 
done in that program so we’ll pile it on... It has been a way that we have been able to 
develop some positive feedback from parents and teachers. I think the thing now is to re- 
educate them as to the value that’s placed on it.. .They appreciate it because it gets done, 
and so they’re ready to move on. Some resent it because they think we’re taking the place 
of the parents’ responsibility and I say, if they can go home and have some quality time at 
home instead of being stressed out and fussing over this homework, that they’re not shirk- 
ing their responsibility. 

In further analyzing the homework situation she pondered if students might not be using their time wisely in 
the classroom and instead opting to do homework at the CLC since they know that they have that block of 
time anyway. She thought that one way to get around this might be to offer exciting learning opportunities 
at the CLC as an incentive for students to complete their homework in class rather than saving it for CLC 
time. 
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(4) Parents’ perceptions. Parents also acknowledge that academic benefits are one of the major 
reasons they send their children to the after-school program and in the same manner as most others, attri- 
bute this to homework completion. In the mid-year evaluation the top three things parents indicated they 
liked about the program included help with homework (receiving 29/35 responses), the additional help their 
children were receiving in reading and math (28/35 responses), and a safe environment and the organized 
activities (each received 25/35 responses). When asked to select the most important reason for enrolling 
their child, they indicated the extra help their children receive in math and reading. Responses at the end of 
the year survey were similar with homework completion ranking number one (26/27 responses). The safe 
environment ranked second (receiving 24/27) and the provision of organized activities and additional help 
in reading and math each received 22 responses. 

Parents do attribute homework completion as an academic benefit in the program - but it seems to be more 
so from the perspective that if children do their homework during the after-school program, then they don’t 
have to “fight” with their children to get their homework done later that evening as one parent explained, 
“Mom is not as grouchy when (her child) comes home due to the fact that the ‘homework fight’ is resolved 
due to most work being completed after school!” Other comments included , “My child is very active and 
trying to do homework with him after school was quite the chore, but since the program started, he com- 
pletes homework while in the program with no problem,” or “She likes to get her homework done.. .We like 
that our evenings have been less stressful without homework too.” It seems that parents feel a sense of 
relief in not having to contend with homework in the evening. One parent who realizes the value that 
teachers placed on homework completion commented that she felt that getting the homework done as 
important so that her son “didn’t get into trouble.” Another parent admitted that the staff in the after-school 
program were more capable of helping her child with homework, especially English homework as she had 
“a hard time remembering all the rules, too.” Parents stated when their children completed their homework 
in the program that meant they had more time together in the evenings to do other types of activities. 

Several parents indicated in their surveys that they believed the primary purpose should be on homework 
completion with one stating that they play a lot and “don’t have time to do their homework.” In fact, one 
parent commented that the reason she was pulling her children out of the program or not re-enrolling them 
was that she believed the children had too much free time and thought the program should have a stronger 
emphasis on schoolwork completion. 

(5) Student grades. Student grades was another indicator used to determine the academic impact the 
program had on students. Grades were tracked in both math and reading. Table IV shows the number of 
students from each school and indicates whether their grades went up, down, or stayed the same. 

An inherent problem in this data is that the basis for grades may be arbitrary (i.e., some may be using 
attendance and homework completion, some may be using unit or chapter tests, etc.) and the fact that 
grading procedures differ from school to school. While curriculum-based measurement (CBM) was used 
and children’s CBM scores increased at both Valley Falls and JCN, the teachers’ grades do not necessarily 
reflect this. CBM is a more effective measure of student progress in reading and math rather than teachers’ 
arbitrary grading. State assessment data would also have been a more accurate measure, but was not 
available for this report. Another problem with these data is that it is difficult to isolate the treatment of the 
CLC from other treatments/strategies that these children encountered during the course of the year. 

Discussion Regarding Academic Benefits 

It is difficult to surmise the reason for the number of students whose grades decreased. The 

McLouth CLC had made a concerted effort to target children most academically in need and then 
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used daily timed math tests as an intervention strategy, and yet in looking at the results shown on 
the grades, McLouth shows the biggest number of students/per students enrolled having the most 
decline in math grades. One needs to question what difference does devoting this segment of time to 
math computation tests and basic facts make when students’ scores in math dropped by a signifi- 
cant number? Except for Perry /Lecompton it appears that all other CLCs had appreciable amounts 
of student decreases in math, while the opposite seems to be true in reading. Better, and more 
reliable, documentation of students’ academic progress is needed. 



Table IV - Grade Data 


Site 


Math 


Up 


Down 


Same 


Reading 


Up 


Down 


Same 


Effingham 




2 


4 


4 




2 


3 


5 


JCN 




3 


5 


7 




6 


2 


7 


McLouth 




1 


8 


9 




8 


3 


7 


Perry-Lecompton 




16 


1 


7 




17 


1 


6 


Valley Falls 




10 


7 


20 




13 


4 


21 



Recommendations: 

• Perhaps in future years more specific measures, such as CBM, might be used across all 
sites so that site coordinators and educators in schools will be able to make a more accu- 
rate determination of students’ academic progress and growth. 

• Perhaps the CLCs need to keep a separate accountability system that measures students’ 
progress and growth rates that can be used in conjunction with the teacher’s assessment of 
student progress. 

There appears to be a paradox regarding homework. While many stakeholder groups had strong 
beliefs concerning how the priority should be placed on homework, yet the research indicates that 
homework doesn’t seem to make an appreciable difference academically with elementary 
children . 12 Parents reported that they appreciated the fact that the CLC’s focused on homework 
completion so that they would not have to “do battle” with their children over homework later in 
the evening and that they could then spend quality family time with their children. Or as one 
principal in relating why parents were so enthusiastic for the program shared, “They don’t have to 
growl around at their kids when they get home about doing their homework, and parents are 
delighted about that one.” She then added, “A list of feedback that I have gotten from my parents is 
they are so delighted that they don’t have to spend it on the homework, instead they can do family 
things.” Another principal made a similar comment in conceding that homework is a “huge struggle 
and it’s kind of nice cause then when they come home they can just hang out the they don’t have to 
have that pressure.” 

On the other hand, many teachers believed that the CLCs were enabling parents not to participate 
in their children’s homework, with some believing that parent supervision of homework completion 
is the only legitimate kind of parent involvement. For instance, in a teacher focus group one teacher 
commented that she “felt like sometimes the parents tried to push too much of that (homework 



completion) on to the school and just totally stepped back from that responsibility.” Other com- 
ments made by teachers concerning this included, “Unfortunately, I think it releases more responsi- 
bility from the parents, as it’s the school that’s taking care of homework needs instead of at home.” 
This person then added that many of those students need this support from school because of the 
possible lack of support at home. Another teacher said, “I feel like the parents have taken a 
backseat and now don’t really bother to work with the kids as much.” And yet another teacher, in 
her response to being asked how the after-school program impacts parent involvement queried, “Is 
it just a babysitting service? Parents feel that they don’t need to spend time working with their child 
at home on academics?” Comments such as this last comment almost illustrate a sense of teacher 
animosity towards the after-school program. 

Comments typical of this seem to indicate that teachers have a limited view of what constitutes 
parent involvement. The literature and research supports an active role by parents in students’ 
achievement, but is often short-sighted in legitimating involvement as constituting other things than 
parents supervising homework completion. While about 30% of the teachers who were surveyed 
said they felt the CLC program impacted parent involvement in a positive way saying things such 
as, “I believe it would allow parents to spend time reading with their children or other activities - 
not spending all evening on work not completed during class time,” 27% believed the program 
negatively impacted parent involvement. These teachers did not consider quality time spent with 
children doing things other than homework, as constituting parent involvement. On the other hand, 
they seemed to appreciate the fact that more children were completing homework. Homework 
completion, it seems, is one of the major determinants for student grades along with attendance. 

The site coordinators and staff reported that homework completion seemed to contribute in a 
positive way to students’ sense of self-esteem - the more that homework was completed, the more 
respect these children seemed to have from their teachers leading to an increased sense of self- 
esteem on the part of the students. 

Recommendations: 

• Perhaps larger discussions are required among day school staff that address the value and 
place of homework in the larger scheme of things (i.e., student understanding and other 
ways to demonstrate understanding other than homework completion). And perhaps 
discussions are needed in schools to expand teachers’ perceptions of what constitutes 
parent involvement. 

• Perhaps discussions with day school staff should also address alternative ways for teachers 
to demonstrate that they value all students beyond completion of homework or that address 
the findings of a meta-analysis on the effectiveness of instructional strategies which places 
a premium on attending to the affective domain. 13 

Consistent with the experience of the CLCs and the homework dilemma, teachers in national 
surveys have reported the following reasons for assigning homework: 

(1) Homework teaches self-discipline. Eighty-eight percent of teachers and principals in one 
study agreed that homework “develops children’s initiative and responsibility”. 14 

(2) Homework is believed to increase student achievement. 

(3) Homework fulfills the expectation of students, parents, and the public. Teachers favor the 
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practice by 95 percent, according to a 1992 Gallup Poll. 15 

(4) Homework increases the length of the school day without increasing the number of hours 
actually spent in school. 

(5) Homework provides an avenue of communication between the school and the parents. 

Contrary to the importance placed on homework completion by adults as constituting a critical 
component of the CLC program, through observations and in querying the children, the homework 
component was clearly disengaging for the children, with some reporting that it was boring and 
others displaying little or no motivation. Through direct observations, it was noted that most of the 
homework assigned to students to complete in the after-school program generally consisted of 
mindless, meaningless, busywork that often consisted of worksheet upon worksheet of math 
problems, lists of spelling words to write and rewrite, and simple question/answer sheet type of 
assignments. For the most part the homework that was assigned was consistent with a “transmis- 
sion” model of education, or to use Paulo Freire’s (a famous Brazilian educator whose life work 
focused on ways to assist students in the production rather than reproduction of knowledge), 
terminology, a “banking process” of education where data deposits are made into the inactive 
mental vaults of students’ brains. 16 The problem with this type of educational process is that it zaps 
teachers and students, alike, of the potential for a meaningful, exciting learning experience. 

Research has shown that homework at the elementary level does not impact students’ achievement 
in school. 17 Other research shows that when the work of school is about atomizing education or 
breaking down learning into minute subskills that there is not a long term impact. In reading 
programs where the focus is primarily on teaching the subskills with homework that primarily 
consists of worksheets or dittos focused on the subskills, students often learn isolated subskills but 
these same students generally are not reading for knowledge, enjoyment, or meaning. Reading 
programs of this nature often commit a fatal error in assuming that the parts add up to the whole. 18 
The same is true for mathematics programs that focus primarily on computation skills and are 
dependent on children memorizing algorithms. These programs don’t enable students to develop 
math power or number sense, the focus of the National Council of Teachers of Mathematics. 

Given this, a paradox exists. Teachers in the sending NEKCLC schools seem to believe that 
homework completion contributes to student achievement. Yet, from observations in these pro- 
grams the majority of the homework assigned is not consistent with the type of skill development 
required of students to perform on the state assessments. While the state assessments are requiring 
that students engage in ill-structured problem-solving, simplistic word lists and math worksheets, 
typical of the assigned homework, instead requires students to find the “right” answer. Students are 
not being asked to engage in the higher order thinking skills that ill-structured problems on the state 
assessments currently require. Teachers in the day program, themselves, need assistance in under- 
standing the nature of ill-structured problems, ways to structure teaching and homework to facili- 
tate students’ ability to do ill-structured problem-solving. But as one principal said, 

That’s like changing the whole paradigm of teaching, you know. Getting it more hands-on 
and I know even when I go into the classrooms I think I have some really good teachers, 
but I’ve had some teachers I’m kind of concerned about because it’s like - OK, we are 
going to do page 28 and 1 through 32.” 




I 
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Recommemdatiora: 



• If one of the goals of the after-school programs is to enable students to achieve at a higher 

rate on the state assessments, then perhaps time spent in the program should focus more so 
on project-based, experiential and hands-on learning where students can be confronted with 
ill-structured problems and challenged to think and apply learning in meaningful contexts. 

While it seems that there are unquestioned assumptions being made about the value of homework 
in these schools, the reality is that a large segment of time is being spent in the CLC program 
devoted to homework completion. Seligson, the founder of the National Institute on Out-of-School 
Time, cautions against the overt emphasis on academic skills, which she says, “can easily slip into 
worksheets and drill if one does not understand that recreation can be educational.” She then adds 
that “we need to be careful about the trend toward seeing after-school programs as ‘homework 
only’ centers, especially given the current emphasis on high stakes testing.” 19 

As observed in all these after-school programs, the homework that students were doing failed to 
spark students’ interests and natural curiosity about learning. If the goal of learning is, as Dewey 
contends, to connect or “mesh” with students’ interests, we need to ask, how are the after-school 
programs enabling this to happen? How does homework spark children’s interest and extend their 
intellect? If one of the goals of the after-school program is to raise student achievement, how do 
staff in the after-school programs justify the large block of time being required to attend to the 
mundane? And finally, where after-school programs have found ways to excite students about 
learning and enable them to practice ill-structured problem-solving (through project-based and 
experiential learning opportunities), how can teachers in the day school program learn and benefit 
from the experience of the after-school program? Regarding the homework dilemma, the program 
coordinator suggested that perhaps during the course of the next year site coordinators could try to 
“look at the quality of what the kids are doing, and what they’re giving up in order to do that.” 

B. Social and Behavioral Benefits 

Because children’s behavioral changes were often determined by how well they interacted in social settings 
with other children, these benefits will be examined together. The program was structured in such a manner 
so as to provide children with ample opportunities to engage in formal and informal activities that assisted 
the development and reinforcement of social and behavioral skills. All programs included a recreational 
component for children to play together in informal settings. All programs also had a designated time slot 
that focused on building healthy relationships. With support from the Kansas Health Foundation, the 
NEKCLC received curriculum materials/supplies and technical assistance from Foundations, Inc., of 
Philadelphia, PA. These materials provided excellent discussion starters to address behavioral and social 
development skills. The nature of the program at most sites, where children were mixed across age- and 
grade-levels, also provided opportunities for them to interact with each other and practice these skills. 

Parents, principals, and site coordinators overwhelmingly believed that the after-school program positively 
impacted children’s social and behavior skills. Two of the five principals interviewed, believed that meeting 
the social/behavioral needs of the children was the greatest benefit of the program. While not perceived as 
the most important benefit of the program, nearly 50% of the parents at mid-year indicated that they 
appreciated the fact that the program provided opportunities for their children to play and interact with 
other children. At the end of the year 66% of the parents responded likewise. Parents indicated that their 
children enjoyed the time spent with other children in the program and the interactions they had with their 



friends and the opportunity to make new friends. Several noted that this interaction had positive effects on 
their children’s sense of self-esteem. Several others commented that they appreciated the opportunity for 
their children to have social contact with older volunteers in the program. 

Being with the children nearly 100% of the time, the site coordinators were able to observe many social and 
behavioral changes in the students. A theme that emerged from their comments was that the after-school 
program engendered a feeling or sense of “family” among the children and the adults/youth assistants in 
the program. They indicated that the children learned how to play and interact with each other, developed 
teamwork skills, and a genuine respect for each other. Concerning specific behavioral changes or skills 
development, comments from Site Coordinators included that the children were more aware of others’ 
feelings, displayed manners (i.e., while eating snacks, opening doors for others), the level of tattling 
decreased and children became more helpful and considerate of each other. One site coordinator commented 
that the program enabled her children to develop anger management skills, to make good choices, to 
develop a sense of humor, and to take responsibility for their actions. 

Teachers had mixed responses concerning the social and behavioral benefits. At mid-year only 43% of the 
teachers responding believed there were positive changes in students’ behavior/social skills. Two teachers 
commented that they believed the program was creating negative behavior in students or feelings of 
frustration. Of the 27 teachers participating in the end-of-the-year review, 13 had positive comments while 
only three responded negatively. Affirmative comments included things such as, “caring, kindness, respect, 
and self control are practiced at CLC” and “very good socialization going on - 1 have witnessed it myself.” 
In a teacher focus group when asked about the social/behavior benefits, one explained 

1 mean that they had a lot of opportunities to use their social skills, a lot of manner oppor- 
tunities with having the snack and that kind of thing, too. But also, they were in a multi- 
grade and so that gave them the opportunity to socialize with older students. 

While one of the principals had indicated a concern with all ages of children being mixed together, most 
other stakeholders believed this “family like” atmosphere was a major benefit of the CLC program. 

Discussion Regarding Social and Behavioral Benefits 

While this structured or unstructured play time was subsumed under the “expanding children’s 
horizons” component in providing children with recreational opportunities, it was quite obvious 
that play time also was quality time for children to experience educational learning experiences as 
well and to develop their motor skills, cooperation, sportsmanship, and healthy relationships as 
well. What was so amazing was how students interacted across age and grade levels - you would 
see the oldest students taking time to play with the younger ones and including everyone in the 
process. This seemed to contribute to other positive interactions the children had during other 
activities in the program. One parent commented to me, that this interaction carried itself into the 
school day and that her son, whom she felt had a lower self-esteem, commented that even during 
the day at school, older students involved in CLC would tell him “hello” when they passed in the 
hall. This little act alone, impacted her son’s self-esteem in a very positive way. So the interaction 
that occurred on the playground carried itself into other aspects of the school and it created a 
feeling of “family” among and between students in the program. 

It was also interesting that students often referred to this time as “recess,” referring to traditional 
language used in the day school program. So often in schools, the physical recreation time can be 



devastating for children, especially those that do not find themselves as physically agile as other 
students (in getting picked last or in not wanting to participate because of lack of skills). But it was 
quite obvious that this feeling did not pervade the CLCs - children cooperated and played together 
quite well. For the most part, staff willingly participated in these activities as well and often 
seemed to use this time as an opportunity to connect with children in meaningful ways. 

C. Youth/Comm unity Enrichment Activities & Safety Concerns 

Providing a safe environment was a high priority for all stakeholders. In the final evaluation, 89% of the 
parents responded that they were appreciative that the after-school program had provided a safe environ- 
ment for their children. Four of the five site coordinators listed this as the greatest benefit to children’s 
involvement in the program with comments such as “allows children a safe place to go with a variety of 
activities.” Principals felt the same way with one believing that the greatest value of the program was 
“having a safe place for kids to be after school.” Another indicated that the only other alternative for some 
of the kids would be “running around town,” so he felt the program certainly addressed a need of providing 
a “safe place for kids to spend two and a half hours.” Many of the teachers and the regional Advisory 
Council also believed that provision of a safe after school environment was a major goal of the program. 

All of the CLC’s were exemplary in the provision of a variety of recreational and cultural enrichment 
activities for children in conjunction with the routine program. While some of the activities included things 
like racing scooters in the gymnasium, checkers tournaments, and project-based activities involving 
puppets, arts and crafts, and technology, others involved activities with participation from outside organi- 
zations. Abilities awareness equipment and materials were provided by the Capper Foundation so students 
could experience life with various disabilities. Independence Inc. followed up with a speaker’s panel visit to 
each of the sites so children could interact with people with disabilities. While students were not involved in 
the needs assessment or planning stages of service-learning projects, to varying degrees most of the CLCs 
participated in community-based service activities. These included involving the children in intergeneration- 
al contact. Children visited nursing homes, assisted living residents and read to the residents, made cards, 
tray favors and door decorations for the senior citizens with whom they interacted. Another site was 
involved in outdoor clean-up activities to commemorate Earth Day. At one site the students performed 
Christmas songs in English, Spanish, and German for the community’s Rotary Club. At another site the 
children prepared a Christmas dinner for their parents and then entertained them with song and poetry. 

With one of the teacher assistants having a baby during the school year, the children at that site sponsored 
a “baby shower.” This same teacher assistant, a professional opera singer, performed an opera for the 
children, who in turn, decorated the facility and made the refreshments. 

In a needs assessment, parents were asked to indicate the types of resources/services and training they felt 
would be beneficial to them. Fifty percent of those responding indicated that they would like parenting ideas 
about how to help their children become better students. Over 30% shared that they would like information 
on parenting skills such as discipline and behavior management, basic training in technology and computer 
applications, and aerobics or fitness classes. In response to meeting these needs, the CLCs co-sponsored a 
parent enrichment activity on “Parenting with Love and Logic.” Over 100 parents attended these three- 
session parenting training classes that happened in three separate locations in the region. To facilitate 
parent involvement, meals were served and babysitting services provided. 

D. Student Resilience and the Needs of At-Risk Students 

One of the project’s goals was to target at-risk students and the Advisory Council spent time during some 
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of their initial meetings discussing how this might happen. While the Council suggested that students 
labeled “at-risk” be given a percentage of the available slots, local sites were given discretion on how to 
handle this. Several teachers when interviewed, shared their concern that children who seemed to need it 
most, were not involved and even though they had encouraged several parents to send their children, these 
parents had decided against it. One principal said that some of her students who needed the program most, 
were not participating and indicated that they ended up not targeting the at-risk students because they 
thought it might appear like a discriminatory practice. At another site, the coordinator indicated that their 
enrollment was 99% at-risk students who had been specifically targeted based on test score data. 

During staff training, the site coordinators were made aware of the resiliency model, based on social 
development theory, that was the basis for program design of the NEKCLC. 20 All activities were purpose- 
ful and thoughtfully planned and implemented at each site as a way to foster resilience in children. While a 
formal component of the program was to “build healthy relationships, the reality, according to the project 
coordinator, was that “everything that happened in the program was building healthy relationships” which 
fostered a sense of resilience in students. She went on to say that she believed the greatest benefits of the 
entire program was the development of relationships between staff and the children. She explained 

1 think that we are there for the kids and I think that’s the greatest benefit. Because, yes, 
we’re supposed to be helping kids meet state standards, etc., but we are the only place 
where there’s an opportunity to relate to kids without a curriculum hanging over our heads. 

So instead of taking kids through curriculum, you can try to get curriculum to go through 
the kids. And our curriculum, in terms of healthy relationships and all of those kinds of 
things, I just think we have a special opportunity to meet the kids where they are, and test 
them, listen to them and value them. 

In examining the many opportunities the CLCs provided to children to foster protective factors, one of the 
major contributors was the development of supportive relationships between the children and the staff - 
i.e., the site coordinators, other adult staff, high school assistants, and community volunteers. While several 
of the sites had either none or two or less adult volunteers, one site had several volunteers that participated 
on a regular basis. Site coordinators believed that one of the greatest benefits of the program for children 
was how it created “a sense of family and belonging.” In observing each of the CLC programs, this was an 
obvious outcome. Site coordinators and their staff were attentive to children’s needs throughout the two and 
half to three hours that children were there; disciplined them, when necessary, in a manner that afforded 
children a sense of dignity and justice; incorporated fun and humor into their programs; and did whatever 
they deemed necessary to build students’ sense of self-esteem. Children participated in numerous opportu- 
nities to explore their environment through all the activities - academic, social, and recreational - that the 
CLCs provided (see Part C - Youth/Community Enrichment Activities & Safety Concerns). 

Seven high school students were involved in the various CLC programs this past year as either volunteers 
or paid student assistants. When asked to, comment on the value of that experience one youth assistant 
responded, “I am satisfied with the experience because it allows me to work with children and share some 
things I have learned in past experiences with them.” The children participating in the program, likewise 
commented that they enjoyed having youth assistants there to help them. Several of the principals indicated 
that they would like to see more mentors involved in the program for the purpose of developing relation- 
ships, with one suggesting middle school, high school, or college student involvement. Another principal 
encouraged the continued practice of involving high school students as mentors in the programs because 
while it had only happened a couple of times during the year (in their particular program), she thought this 
would be an excellent thing to continue because, “The children need those role models, “ adding “It’s 



always kind of gratifying to have those little kids look up to them.” 

Concerning the self-efficacy variable, 43% of the parents surveyed (mid-year) indicated that their children 
saw themselves as more successful in the classroom. Several also indicated that because children were 
working on and completing homework in the program, this contributed to less family stress and better 
quality family time in the evenings. One principal commented that she believed the greatest value of the 
program was the “self-confidence, the self-esteem the children have gotten from this.” At the mid-year 
evaluation, 65% of the teachers reported positive impacts on students’ self-concept or feeling of success. 
Several commented that their students were excited to show off what they’ve done and that they feel better 
about themselves because they are caught up on their work or come better prepared for school. One said, 
“when work is completed, there is definitely a notable behavior displaying their pride in completing then- 
work.” Another commented that the CLC has “made a huge impact on my students’ self-concept.” Teach- 
ers’ responses at the end of the year were overwhelmingly positive concerning the program’s impact on 
students’ level of self-esteem and confidence. There were several who either disagreed or had no opinion. 
One said the after-school program wasn’t “detrimental” and another commented, “Hard to tell if then- 
improved self-concept is due to after-school program or my own teaching.” 

Discussion Regarding Student Resilience 

School children are faced with countless risks that threaten their personal, social, and academic 
well-being. In spite of these, many children are able to demonstrate remarkable resilience, and are 
able to succeed. Many researchers, including McMillan and Reed 21 and Werner 22 discuss the many 
protective factors that foster resiliency. The challenge for schools is to maximize these protective 
factors as they design and implement programs that foster resiliency in children. Table V lists 
many of these factors. An analysis of the NEKCLC programs reveals that of the 28 protective 
factors listed, there is sufficient evidence to demonstrate that at least 17 of these factors (those with 
the asterisk) were met on a routine basis in all of the after-school programs. 



Table V - Protective Factors That Promote Resiliency 



* Admiration and supportive 
relationships with at least one 
adult 

^Belief that goals are achievable 
Caring parents 
Clear long-term goals 
College preparatory plans 
Enhanced opportunities at major 
life transitions 

Feelings of personal control over 
one’s life 

Having experienced lessons (“re- 
ality checks” ) that reverse the 
allure of risk behaviors 



Good health 
*Low family stress 
*Opportunities to explore one’s 
environment 

Perceptions that dysfunctional 
home environments do not hin 
der academic success 
^Personal responsibility 
^Presence of supportive adults 
^Schools that emphasize involve- 
ment and belonging 
* Self-efficacy (in various domains) 
*Self-esteem 



*Sense of community belonging 
*Sense of dignity 
*Sense of humor 

* Sense of justice 
Spiritual belonging 

* Sports participation 
^Supportive school personnel 
^Unconditional love 
*Using time positively and 

productively 

Virtue 

Well-developed maturity 
Optimism about one’s future 
*Met by the CLC 



E. Fidelity to the Program Model and Criteria for Quality Programs 

One of the primary questions of the evaluation was to what degree the CLCs implemented the design of the 
model as it was conceptualized in the proposal. In addition to assessing fidelity, it was also important to 
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determine the site coordinators’ opinions regarding the value these components had in the overall design of 
the after-school programs. Respondents were given a survey with a Likert scale of 1-5, with a one rating 
indicating either no implementation and not valuable, a three meaning the component was somewhat being 
implemented and perceived as having neutral value to the program, and a five meaning complete implemen- 
tation and extremely valuable and a necessary component of after-school programs. Table VI shows the 
mean value assigned to each component (One of the sites inadvertently left one page blank). 



Table VI - Mean Scores Concerning Fidelity to the Model 


Component 


IMPL 

Rating 


Comments 


VALUE 

Rating 


Comments 


Component 1: Strengthening academic skills 


Direct instruc- 
tion 


2.6 


This was reported as no use, to 
sporadic use, to consistent use 
w/certain children. 


3.4 


One reported that this seemed like “overkill”; 
another was “not sure how many would bene- 
fit;” another said she was not adequately 
trained; and one indicated it worked well. 


Learning strat- 
egies and study 
skills 


3.4 


While one said it was never thor- 
oughly explained, others indi- 
cated using various strategies and 
study skills through games, com- 
puters, etc. 


4.75 


All reported that they wanted to teach more 
study skills. 


Peer/cross-age 

tutoring 


4 


All reported they used this to a 
certain extent and that children 
enjoyed it. Where it wasn’t used 
as readily was where the children 
were academically poor readers 
or where the kids bullied others. 


4.6 


They all indicated that it was extremely im- 
portant, except one who gave it a 3 rating. 


Service-learn- 

ing 


3.2 


Most CLCs engaged in a variety 
of community-service activities, 
but not necessarily service-learn- 
ing. 


4.4 


They all reported this as valuable and said it 
was necessary for children to be a part of, and 
feel responsibility toward the community. 


Hands-on, ex- 
periential, pro- 
ject/based 
learning 


4.6 


CLCs reported many opportuni- 
ties and activities used - experi- 
ments, inventions, crafts, play 
writing, creative movement, built 
an obstacle course, etc. 


4.8 


One said, “Students don’t want to sit and do 
seat work another VA hours during CLC.” 
Others highly endorsed this type of learning 
with one commenting that, “Quiet learning 
was not in our vocabulary.” 


Access to 
school libraries 


4.4 


Effingham reported no access - 
they used the public library one 
day/week; Perry/Lecompton felt 
they had restricted use). 


4.4 
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Homework as- 
sistance 


5 


While all devoted a large time 
block to homework, McLouth 
spent up to an hour/day. 


5 


This was rated a 5 by all coordinators with 
comments such as “95% of my students were 
there for the academics,” “parents and chil- 
dren appeared to appreciate the work being 
completed,” and “it is important to follow 
through the classroom teacher’s requests to 
have a successful collaboration with staff.” 


Community/ 
VISTA volun- 
teer reading 
assistance 


2.6 


One CLC had numerous volun- 
teers providing assistance, an- 
other had four HS students, the 
others reported they needed to 
work on this component. 


4.4 


All valued this as a 5 but one that gave it a 2 
rating commented, “I need to do a lot more 
next year.” 


Component 2: Building healthy relationships 


Integrated 
health/social 
serv prog to 
foster resilience 


3.5 


Coordinators reported guest 
speakers- presented info on to- 
bacco use, alcohol, and drugs; 
Abilities Awareness activities. 


4.75 


All believed that this was an extremely impor- 
tant component and that ‘"the earlier we can 
reach kids on these topics the better.” 


Opportunities 
for children to 
address or 
solve problems 
and increase 
sense of self- 
esteem 


3.75 


All reported many ways of teach- 
ing problem-solving skills by 
pulling a problem out of a jar and 
solving it on the spot; drama and 
puppets; teamwork; daily social- 
izing, and teaching anger man- 
agement techniques 


5 


All rated this as a 5 with comments such as it 
is important to develop a positive self-concept 
and for all to have a need for self-worth 


Component3: Widening children’s horizons 


Provision of 
recreational 
activities 


4.75 


Many opportunities were pro- 
vided-structured/ unstructured 
time outside, organized games, 
“snowball fights” with newspa- 
pers, dancing, strategy games, 
and an obstacle course. 


5 


All gave this a 5 saying that this teaches coor- 
dination, social skills, sharing, strategy, and 
the ability to play and get along with a variety 
of age groups. One commented that “after we 
play and laugh the children are ready to 
study.” And another said “it helped so much 
in developing healthy relationships.” 


Provision of 
cultural activi- 
ties 


3.75 


One site reported a 5, while the 
others 2s and a 4, indicating that 
while some activities occurred 
there was a need for more. Some 
invited in speakers while others 
used literature. 


5 


All reported the value as a 5 saying it is criti- 
cal for children to be aware of all cultures, 
especially where there is little diversity. 


Provision of 
enrichment 
activities (mu- 
sic/foreign lan- 
guage, etc.) 


3.25 


All indicated that they incorpo- 
rated music and/or dance; two 
offered minimal foreign language 
in Spanish and French. 


4.75 


They believed that enhancing the curriculum 
was important in sustaining student interest. 
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Provision of 
technology-re- 
lated activities 


3.4 


Three reported using or having 
access to computers daily. One 
said, “Not this year.” Another 
said that while they had comput- 
ers, the technology was old and 
outdated. 


4.2 


They felt the value of technology was exposing 
students to a variety of learning tools. 


Involvement in 
community- 
based youth 
organizations 
(i.e., Boy/Girl 
Scouts, etc.) 


2.2 


While one rated this a 5 having 
several of her students participate 
in sports and youth organiza- 
tions, 3 rated this as land one a 
3. 


3.8 


They all, except one, indicated it was impor- 
tant saying that it would be neat to have differ- 
ent organizations working with the CLCs to 
broaden concepts and ideas, or to work on 
large, special projects. 


Component 4: Establishing Parent Resource Centers 


Parents partici- 
pate in or vol- 
unteer in chil- 
dren’s activities 


2.6 


This was rated rather low with 
coordinators saying that they had 
few or no parents who 
volunteered in the program. 


4 


While they all felt it would be helpful to have 
parents, some questioned how realistic that is 
with working parents. 


Parenting 

knowledge/skill 

development 


2.8 


Rated rather low again, one of 
the only activities was Love & 
Logic training for parents, of 
which not too many CLC parents 
attended. 


4.4 


Three of the four indicated that it was impor- 
tant to offer training for parents in more/better 
parenting skills. 


Technology 
skills training 


1 


All reported that this was not 
implemented. 


4 


Four felt that this was something that should 
be offered in the future. 



Six of the 18 components identified as important for after-school programs received less than a three rating, 
meaning that they were somewhere on the scale between not implemented and somewhat implemented. Of 
these six items, only two were rated as less than a four concerning the value of the component - with direct 
instruction receiving ratings of 2.6 and 3.4 and involvement in community-based youth organizations 
receiving a 2.2 and 3.8. The lowest value for any one item was given to direct instruction, indicating mixed 
feelings about including direct instruction as part of the program model. The program coordinator indicated 
that direct instruction was the “biggest problem in terms of getting it implemented.” She went on to say that 
she believed things would be different the next year with the training her staff was receiving and the 
thinking that direct instruction techniques could be incorporated “at any given moment, teachable moment” 
to direct students’ attention to the lesson being taught. 

Commenting on the service-learning component that received ratings of 3.2 and 4.4, the program coordina- 
tor commented that it had been minimally implemented this year and that it really had entailed community 
service rather than service-learning. But she also commented that steps were being taken to introduce staff 
to broader perspectives on service-learning - how it could be used to reinforce learning, how the children 
should be involved in determining the learning needs and the planning of the activities. She added, 

The thing that I think is very valuable, if first the kids are involved in planning. If you’re 
just going through the motions of community service, that serves one purpose. It doesn’t 
let you capitalize on the way that it takes them through the problem solving and the plan- 
ning and the organizing. To me, that’s where the learning comes from. 
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The only program component that received a five for both implementation and value was homework 
assistance, indicating that the CLCs were devoting sufficient to considerable time for homework completion 
and site coordinators believing that it was an important and necessary component. At one of the CLCs up 
to one hour was being devoted to homework completion. 

In responses to questions concerning the implementation of the Parent Resource Center component, which 
received the lowest rating overall regarding implementation, the program coordinator indicated that while 
they had a long way to go they had made “an excellent start with Love and Logic” training. She said that 
for the next year she had purchased resources on parenting skills so that each site has the same collection of 
materials and resources. She also related that dates were already set for the following year to do more 
training in “Love and Logic.” Concerning the provision of technology skills training, she indicated that she 
planned to contact agencies in the region who provide such training and use the CLC as an opportunity to 
make parents aware of these learning opportunities. 

In examining the data, generated from the site coordinators’ survey, from another perspective - looking at 
the variance between the ideal (the value placed on the components) and the reality (the degree of imple- 
mentation) - there is significant variation (a difference of at least 3 or 4 points) with eight components at 
several of the sites. Table VII summarizes where this variance is occurring and indicates possibly more 
time needed to work with staff, or an examination of strategies that could be used concerning implementa- 
tion of these components. 



Table VII - Variance Scores 


Component 


Site 


Variance Differential 


Direct instruction 


Valley Falls 


4 


Comm unity/ VI ST A volunteer reading assistance 


McLouth 


4 




Perry/Lecompton 


3 


Provision of cultural activities 


McLouth 


3 




JCN 


3 


Provision of enrichment activities 


McLouth 


3 


Involvement in community-based youth organizations 


McLouth 


4 




Valley Falls 


3 


Parents participate/volunteer in children’s activities 


Effingham 


3 


Opportunities for parenting knowledge/skill development 


Perry/Lecompton 


3 


Opportunities to receive technology skills training 


McLouth 


4 




Valley Falls 


4 




JCN 


4 




Perry/Lecompton 


3 



Discussion Regarding Program Fidelity 

Examining the fidelity of the implemented program to the conceptualized program is one way to 
assess the correspondence of the program as documented to the program as delivered. It is also an 
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important way to maintain quality control in the program. But the challenge, as in the case of these 
programs, is how to maintain program standardization while also maintaining a degree of flexibil- 
ity and autonomy, given the fact that each program resided in a different school district, each with 
their own goals. Yet if one hopes to attain the desired and expected outcomes, then program fidelity 
is important. One way to assure fidelity while respecting the need for adaptability is to identify 
“core” components that single out the parts of the program that cannot change if the desired 
outcomes are to be achieved. This is why this evaluator believed it was important to not only assess 
the degree of implementation, but to understand how site coordinators valued these various pro- 
gram components, as the values one holds about these certainly impacts the implementation. 

In examining the data presented in this section it is clear, that during the first year of the NEKCLC 
after-school programs, there has been a relatively high degree of fidelity to the model. While not all 
components have been implemented to the same degree as others, an attempt is being made to move 
in that direction. Looking at the process of implementation has provided keen insights into the 
kinds of professional development that are required to carry out the program developer’s intent. 
Such is the case with direct instruction - plans have been made to provide further training to site 
coordinators to give them strategies and alternative ways to incorporate direct instruction into the 
daily program. With other less implemented components, such as the Parent Resource Centers, 
plans are in the making to bring this component up to par as well. 

Recommendation: 

° Perhaps an element in future professional development focused on implementation of 

program components, time might be spent examining perceived values of these components 
as a starting point to understanding where and why resistance to implementation might 
occur. 

F. Comparison to Quality Criteria 

The National School-Age Care Alliance (NS AC A), a professional membership organization of school-age 
providers, trainers, and advocates, has established program standards for quality school-age child care that 
describe best practices. The standards are organized into six categories: human relationships, indoor 
environment, outdoor environment, activities, safety/health/nutrition, and administration. Approximately 
one third of the standards fall under human relationships. The standards use a four point Likert scale with 
one meaning no evidence to support this, two is happening to a limited degree, three means happening quite 
regularly, and four is occurring all the time. As a way to analyze observation data, all sites were evaluated 
against these standards. The 16 standards that comprise administration were used as a basis to organize the 
site coordinators’ survey and the program coordinator interview. Table VIII presents a summary of the 
evaluator’s impressions of the NEKCLC after-school programs judged against the NSACA standards in 
five of the six categories. 

It is important to note that some of the standards ask questions about events that were not observable 
during any of the observations, such as Standard 6 (d) - When problems occur, children often try to 
discuss their differences and work out a solution. Conflicts rarely occurred at any of the CLC sites during 
observations. While several of the activities asked for in the standards may not have occurred during those 
three visits, this is not to say that these activities never occurred. 
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Table VIII - Comparison of NEKCLC’s to NSACA Standards 


Standards 


Items/ 
Total Points 


Effingham 


JJCN 


McLouth 


Perry/ 

Lecompton 


Valley Falls 


Human Relationships 


30/120 


92/100-92% 


94/100-94% 


92/108-85% 


69/104-66% 


95/100—95% 


Indoor Environment 


8/32 


13/32-41% 


26/32-81% 


29/32-90% 


23/28-82% 


29/32-91% 


Outdoor Environment 


4/16 


16/16-100% 


16/16-100% 


14/16-88% 


14/16-88% 


16/16-100% 


Activities 


12/48 


42/44-99% 


43/44-98% 


36/44-82% 


34 / 44 - 770/0 


44/44-100% 


Safety, Health & Nu- 
trition 


20/80 


58/60-97% 


60/60-100% 


62/64-97% 


54/56-96% 


64/64-100% 



In explaining the above ratings, the second column lists the number of items and the number of total points 
possible, so if there are 30 items and the top rating is four points, then the number of possible points for 
that particular category of standards would be 120. Under the program site columns is listed the total 
number of points from the ratings given during the observations as compared to the items that were rated. 
Consequently, under column three in the standards category human relationships, 25 of 30 sub-standards 
were observed in this category at the Effingham site. The total points possible were 100 and Effingham 
received 92 or 92% in this particular category. Under the human relationships category, McLouth received 
92 out of a possible 108 points or achieved an 85% rating overall for this category; the Perry/Lecompton 
site was rated at 69/104 or a 66% rating in this category. 

Discussion Concerning Quality Criteria Comparisons 

An analysis of these ratings indicate that some of the CLC’s rated higher in several of the stan- 
dards categories than the others. For instance, the Perry-Lecompton CLC rated the lowest in the 
human relationships category which includes sub-standards that address how well staff and 
children interact in all activities and staff interactions among themselves. Observations of this site 
confirm these lower ratings - use of an inexperienced high school student teaching assistant with 
the youngest and most needy group concerning literacy development; the lack of interaction 
between staff and children during the recreational component; separating children into grade- 
specific classrooms where no interaction among and between children occurred as did at the other 
sites; and the site coordinator’s communication style with her staff using a walkie-talkie. 

Effingham rated considerably lower on the indoor environment standard. This is not an indictment 
of the program at Effingham, which received high ratings in all other categories, but rather shows 
the limitation of the space provided for the CLC program. The average rating for all CLC pro- 
grams across all standards areas was 90% which is a high rating. The average rating of individual 
categories of the standards across all standards areas was 89%, another high rating - but with 
room for improvement. 

The final set of standards NSACA covers Administration Standards and includes 16 separate indicators 
that are briefly summarized below: 

1. Stafi/child ratios sufficient to meet needs of children. The number of children participating in 
the program varied from site to site. Effingham had the fewest number of children with 1 1 regu- 



larly attending and Valley Falls had the greatest number with up to 41 attendees. In addition to 
paid staff, Valley Falls also had access to several volunteers who helped maintain the 
teacher/student ratio within a reasonable (and acceptable) amount. The Executive Director of the 
agency overseeing this project has expressed concerns about the low number of attendees and has 
encouraged the project director to work with her staff in more intensive recruitment procedures. 

Recommendation: 

• Perhaps the Advisory Council could assist in a PR campaign to ensure higher number of 
children participating in future years. 

Children supervised at all times. Children were more than adequately supervised at all times in 
all programs. It was unanimous among all stakeholder groups that the CLCs provided a place 
“with supervised care and structure.” 

Support for family involvement. While the formation of Parent Resource Centers were written 
into the program design with the intent to foster parent involvement, the CLCs fell short of meeting 
this goal. While a “Love & Logic” parenting course was offered that was extremely well attended 
by parents throughout the region, not as many CLC parents availed themselves of this opportunity 
as had been hoped. On the other hand, the CLCs were quite effective at supporting parent involve- 
ment from the perspective of homework completion - parents reported feelings of relief in not 
having to fight with their children over homework and believing it freed up time to engage in 
quality family activities. 

Recommendation: 

• Perhaps a more concerted effort should be made to look for alternative ways to actively 
involve parents. One principal suggested that a series of focus groups be held with parents 
on topics of concern. Parents also suggested other types of courses on parenting skills. 
Another suggestion is to consider how the CLCs can broker information to parents about 
educational opportunities in the region. 

Staff, families, and schools communicate with each other. Communication between the day 
school and after-school programs for most sites was quite adequate with 28/37 teachers rating 
communication as either a four or five (5=extremely good) at the mid-year evaluation and 18/27 
rating it as a four or five at the end of the year. But overall satisfaction with communication 
declined ten percent by the end of the year, with seven teachers rating it as either a two or three (1= 
communication not good at all). One of the principals, whose children were bused from their school 
to another elementary school to attend their CLC program, indicated that communication was the 
biggest problem they had encountered. He indicated that while measures had been put into place to 
facilitate the situation, these efforts had been abandoned by the end of the year. 

Various means were used to communicate (i.e., verbal, notes, in person, visits, etc.) with a home- 
work tracker/folder that was passed between the teacher and the site coordinator as the most 
prevalent method. Eleven teachers reported communicating with their site coordinator on a weekly 
basis and 15 communicating every other week, very seldom, or only when necessary. One teacher 
suggested that site coordinators try e-mailing as a more effective way to facilitate communication. 



5. Program builds links to community. Linkages to the community were established in several ways 
- in visiting community-based organizations and facilities, performances for community groups, 
involvement in clean-up projects that lent visibility in the community, and through the involvement 
of community volunteers. At the present time, no linkages were happening with other community- 
based youth organizations, i.e., Girl and Boy Scouts or 4H, yet attendance in the after-school 
programs dropped off in the spring with students joining structured sports teams. One site coordi- 
nator suggested that they meet with these groups to collaborate on how they could both meet 
students’ needs. 

Recommendation: 

° Perhaps in future years the Advisory Committee might work with the site coordinators, the 
program coordinator, and site principals to generate other ideas to create tighter commu- 
nity linkages. 

6. Indoor space meets needs of staff. Storage space seemed to be an ongoing concern for most site 
coordinators. The Effingham site had extremely limited space for implementing a quality program. 
Things eased up a little once teachers began to allow their rooms to be used for the academic 
component in the program. The site coordinator and principal at Perry/Lecompton both mentioned 
space issues, even though the program occupied numerous classrooms, the cafeteria, and occasion- 
ally the school library. 

7. Outdoor space is sufficient. All programs had more than adequate space for outdoor activities, 
being able to have the children use the playground and the equipment from the regular school 
program. 

8. Staff and children plan activities together. While children were actively involved in a variety of 
activities, there was no evidence, of which the evaluator is aware of joint planning between staff 
and children, other than several of the activities at various sites, such as the decorations for the 
opera at Valley Falls. If service- learning takes on a more prominent role in the CLC program, the 
potential for involving students in decisions about the program and activities has much more 
potential. Subsumed under this standard are “activities that reflect the culture of the families in the 
program and the broad diversity of human experience.” Through the Capper Foundation, children’s 
consciousness was raised concerning disability. Students had opportunities to learn about disabili- 
ties and interact with persons who had disabilities. Several of the CLCs exposed their students to 
other cultures - but this was not common across all program sites. 

9. Polices and procedures concerning safety needs. Procedures were in place at all CLCs to handle 
safety issues. Each program has a check-out system in place to prevent unauthorized people from 
picking up children at the end of the day. Those programs that transport children have established 
policies to transport them safely that complies with all legal requirements for vehicles and drivers. 

10. Policies and procedures to enhance health. Similarly, policies and procedures were formulated 
that addressed these issues. All staff were trained to respond to accidents and emergencies. 

11. Staff professionally qualified. The program coordinator was an extremely qualified educator 
having had 23 years experience working in Title I programs and having coordinated her district 
program. While coordinating a program of this magnitude was a new experience for her, she rose 
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to the challenge and was extremely well-received by five of the six principals involved in the 
program. In reflecting on the project coordinator’s effectiveness in managing the program, one 
principal commented, “I feel that it has good leadership.” Another principal responded that while 
the project coordinator must have been initially overwhelmed, she said, “I really think she has done 
an exceptional job.” She then added, 

She communicates with me frequently. She must work 80 hours a week. You know she 
really gets out to the sites, she actually did some of the CBM testing herself. 

Qualified staff were hired for each program. While all were not certified teachers, i.e., one having a 
social work background and another having teaching experience in a religious setting, all rose to 
the challenge and implemented programs that ranged from fairly successful to extremely success- 
ful. 

12. All staff provided with training. Prior to the program beginning and intermittently throughout the 
year, staff had opportunities to participate in various professional development opportunities. Four 
of the five site coordinators believed that the training was adequate and made comments such as, 
“Training has been sufficient, but I would like more hands-on type training for learning strategies 
using experiments, crafts, etc.,” and “I don’t think any amount of training would have totally 
prepared us for the first year but I do feel we had opportunities to learn and prepare.” One site 
coordinator indicated that she would have liked more background information on building resil- 
iency in children. Another remarked that the training was so compact, “that it left some (like me) in 
a fog.” She went on to say that the training was good, but she would have preferred “more time to 
process and put into action what is learned before new material is presented.” 

13. Training needs of staff assessed and training is relevant to job. Because the goal of training was 
to provide staff with sufficient background knowledge and information in implementing the 
NEKCLC program design, there was not a lot of room and time to provide training in other areas 
outside of the model. But on the other hand, all training was relevant to site coordinators being able 
to implement the model at their schools. The evaluator is not aware of an assessment being used to 
determine training needs. 

14. Staff receive appropriate support. Site coordinators, for the most part, felt more than adequately 
supported at their sites. One site coordinator commented, “I feel the CLC program is supported in 
whatever we are doing.” Another said, “I see the coordination as always being good. The questions 
in the beginning were answered and provided leadership to move forward.” Another coordinator 
also commented that she appreciated Melissa’s assistance (the project coordinator’s secretary) in 
putting together all the government forms and keeping the paperwork in order. 

In responding specifically to the support that was given by the project coordinator, the responses 
were overwhelmingly positive. One explained in detail, 

Sharon was just like all the site coordinators this year. We were all wandering in unfamil- 
iar territory with this grant. She explained everything she knew to the best of her abilities. 

Sharon was very supportive of us and our ideas as well. Once our programs were started I 
saw a lot less of her, which I was fine with. If I ever had questions she was more than 
happy to help me. I guess in my own sense I didn’t rely too heavily on her because I felt 
comfortable. 
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Others commented that the project coordinator “empowered” them to work out difficulties, but 
“was still with us in consulting.” And another said that “she has always supported me with any 
ideas I wanted to try.” 

In examining the role of principals in supporting the CLC’s efforts, principals had several interest- 
ing comments. Two indicated that support meant making sure they had all the things they needed, 
like supplies and transportation. Another said it meant reassuring local childcare providers that the 
CLC program was not their competition. Another principal believed that support consisted of 
following through on the discipline, supporting the CLC teachers, and being visible in the program 
- but then added that this had not happened as often as she would have liked. Three of the princi- 
pals believed that support meant connecting with relevant stakeholders in the district and communi- 
cating with them about the program. One said that the role of the principal is to be a “good listener, 
and listen to other different factors that are influenced by the after-school program.. .the parents, 
the students, the staff", and then the regular education staff.” This principal added, 

Asa principal, 1 need to work with the site coordinator and provide whatever guidance as 
far as a little background knowledge about some of the students and the parents, if they 
need to have me support them if there is a problem that has come up with parents or with 
other students, or whatever, that’s another thing that I need to do. 

15. Administration provides sound management of the program. The sub-standards under this 
standard include providing a sufficient budget to support the program’s goals. The budget was 
evidently sufficient as there was a large carryover that was requested by the project coordinator. 
Consequently, sites had sufficient materials and supplies and personnel to carry out all necessary 
responsibilities in the program. The program coordinator became an expert at networking and 
locating resources that had minimal cost to the program. A regional Advisory Council was formed 
early in the program and met on a regular basis throughout the year to provide long-term planning 
and direction in daily decision-making, especially on issues such as recruitment policies faced by 
the sites. The evaluator met intermittently throughout the year with the project coordinator to plan 
and coordinate an ongoing formal evaluation process. A mid-year evaluation provided useful data 
to the program staff to assess program progress and use the data for continuous improvement 
purposes. 

16. Policies and procedures responsive to needs of children/families. The program’s philosophy and 
goals are laid out in the program design which has been the basis for the operation of the program. 
Every decision that has been made about the program is a reflection on the model. While all 
components have not had an equally successful implementation, all components are in the process 
of being implemented. The program has been affordable to families, charging them $1.00 per day 
per child during the first year of operation. For those parents who can not afford the fees, SRS has 
provided subsidies to cover cost for these children. The program’s hours have been flexible based 
on parents’ needs. While this has sometimes been detrimental in children receiving less benefits 
than others, due to a shortened attendance period, this adjustment meets parents’ needs. All of the 
sites have enrolled children that have unique and special needs with one site coordinator indicating 
that her program consisted of over 95% at-risk children. One program initially experienced diffi- 
culty in trying to accommodate children who had severe behavioral and emotional needs. 

Recommendation : 
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° Perhaps in future years, alternative strategies may be required to assist these programs in 
meeting the needs of children identified with special needs. 

Discussion of NSACA Administrative Standards 

In summary of the NSACA Administration Standards addressed in the NEKCLC, out of a possible 
point value of 256, the evaluator has rated this category as 232 or given the regional program a 
91% rating. In relationship to the NSACA standards, areas that might be examined include such 
things as how the CLC staff assists families in communicating and working more effectively with 
the schools, how the CLC program notifies parents about community resources to meet the needs 
of children and their families; the space issue (especially at the Effingham site) and storage con- 
cerns for several of the site coordinators; and how to work more effectively with students identified 
with special needs. 

Recommendation: 

• The NSACA standards provide a valuable tool for evaluating after-school programs and 
perhaps could be used as a guide for site coordinators and their staff to engage in self- 
reflection on their programs and a beginning point to determine where program improve- 
ment is required. 

G. CLCs as Center of the Community 

The answer to this question has been addressed throughout this evaluation report. During the initial year of 
operation the priority seemed to lie with getting the program up and running so that the children would be 
the primary beneficiaries. Attempts were made to involve the larger community, i.e, parents and community 
members through several means such as dinners and performances for community members and families, 
open-houses where families were invited to attend the CLC programs, “Love and Logic” parenting skills 
training, surveys to ascertain parent needs, community-based service to organizations within the communi- 
ties, and provision of volunteering opportunities for community members. 

Recommendation : 

• Now that the program is well established, energy needs to focus on this component as well. 

H. Collaboration with School Staff and Community Organizations 

The answer to this question is also embedded in this evaluation report. Specifically see numbers 4 and 5 
under the Administration Standards component in F (Comparison to Quality Criteria) that illustrates the 
quality of the communication process between the CLCs and the school staff and how the CLCs interfaced 
with community and community-based organizations. 

I. Satisfaction/Impact of the CLC 

Writing the grant and successfully winning the money to support five after-school programs seems like the 
easy part of this process. The far more difficult aspect is the actual implementation of the program - hiring 
and training the staff, securing adequate resources/materials, ensuring commitment from the schools - the 
teachers, principals and support staff, gaining parents’ confidence to send their children to this program, 
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and finally, providing a program in which children want to willingly participate. And yet, in the span of one 
short year, all this happened with well overlOO children from northeast Kansas being able to experience 
this educationally sound and meaningful opportunity. 

Based on the findings enumerated in this evaluation report, the obvious conclusion is that the vast majority 
of all stakeholder groups were extremely satisfied with theNEKCLC program and the individual sites. 
Satisfaction from the program was based on all the needs the program met - from providing a safe place 
where students could go after the school day; to the academic assistance they received, especially with their 
homework; to the way the program engendered in children an increase in their self-esteem; and given the 
fact that the program exemplified a community of children where they were able to interact and support 
each other across age and grade levels. Perhaps a statement made by one parent best sums up the general 
feeling that stakeholders had about the program which was, “We feel this program is the best thing that has 
happened relating to school in our community!” 

While several problems and issues emerged over the course of the year and while there were several 
teachers who did not feel the program was a worthwhile activity, or several parents who decided that the 
program did not meet their needs, or several children who said they preferred not to attend, these comprised 
an insignificant number given the numbers of persons (i.e., parents, children, teachers, principals, Advisory 
Council members, community volunteers, and program staff) who adamantly supported the program, 
expressed their satisfaction and gratitude for the program, and believed that the program accomplished 
what it set out to do. 

J. Conclusions and Implications 

In looking at the best practices that occurred across NEKCLC program sites it seems that certain charac- 
teristics emerged that contributes to the quality of the CLC program design. Given the findings, discussion, 
and recommendations shared in this evaluation report, the following constitute, what this evaluator believes 
to be characteristics of quality after-school programs. 

1 . A program model grounded in social development theory and resilience research and 
deliberate decisions made about the program activities and thoughts about how each 
contributes to the program model goals and increases students’ sense of resilience. In 
addition to having a sound model, programs must provide the necessary professional 
development and support to ensure that the model components are understood and are 
being implemented to the fullest extent possible. 

2. If supplemental programs like these are going to be successful it will be necessary to 
secure teacher input and involvement in the design and implementation of the program to 
ensure buy-in and support. Teachers also need to be helped to see how after-school pro- 
grams can assist them in the teaching process, otherwise they may feel threatened when the 
program seems to be having a degree of success with students they are unable to reach. 
Teachers overt resistance can actually sabotage program outcomes. 

3. Time built into the model at the beginning of the after-school program period where all 
children can process their school day - sharing things about their day and having time to 
debrief what occurred during the day. This provides an excellent time to develop interper- 
sonal and social skills as well as skills of communication - both listening and speaking. It 
also gives the children an opportunity to have their voices heard and contributes to building 



healthy relationships among and between children. Most importantly it may provide the 
kind of “family time” that so many children today are missing in their lives. 

4. An adequate amount of time to spend doing recreational activities where children can play 
with each other in structured and unstructured activities. It is important for staff to partici- 
pate with the children as well, as this provides an additional opportunity to attend to chil- 
dren’s personal and social needs. 

5. Going back to the “one room schoolhouse” concept, those sites that organized their pro- 
grams around mixed grade levels seemed to engender more opportunities for children to 
learn from each other in formal and informal settings. Peer and cross-age tutoring naturally 
happened in these programs. When children are isolated by age or grade level it seems that 
there are missed opportunities for this type of rich interaction and potential for learning 
and individual growth. This is one of the values of being an after-school program - not 
having to look like the traditional school setting. 

6. Hands-on, experiential, and project-based learning seem to really motivate and excite the 
children. They are learning and don’t even realize it. Through learning of this nature, 
hopefully the after-school programs will provide a model to the regular school program 
that children can learn and have fun at the same time. Learning doesn’t have to mean work 
that is a drudgery and unjoyful! 

7. There are many ways to reinforce basic math and reading skills and concepts, as has been 
shown through the various after-school program activities, without having to do laborious 
worksheets and “drill and kill” practices. While these certainly have their place, we are 
obligated as educators to find ways to extend learning in alternative ways. Consequently, 
finding a balance between time spent on homework completion and enrichment activities is 
a necessity. 

8. Involving the community (through volunteers, service-learning, or community service) has 
benefits in connecting the children to their communities and can have educational benefits 
when children are shown how their learning can be applied in authentic and practical ways. 
Service-learning is an excellent way to accomplish both community involvement while it 
reinforces learning at the same time. 

9. Quality programs must have an evaluation component built-in to the program that facili- 
tates an evaluation of the ecology of the program - the entire process and all its various 
component parts. Such an evaluation will provide summative data so that planners and 
implementers are aware of program outcomes and the degree of success in meeting these. 
But more importantly, this type of an evaluation process will also provide valuable 
information that will assist planners and implementers in understanding the program and 
give insights into how to continually improve the program. 

10. Before after-school programs address the issue of sustainability, they first must ensure a 
quality program worthy of sustaining. If local stakeholders believe the program is worth- 
while, they will find it necessary to generate the monies to continue the support. 
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